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ABSTRACT 



The evaluation of World Wide Web resources has become a 



critical information skill for students. Because of the uniqueness of these 
resources, new tools are required that assess not only the content validity 
and mechanical functionality of a Web site, but also its motivational 
quality. This paper describes the development and validation of a set of Web 
evaluation tools, the Website Motivational Analysis Checklist (WebMAC) , 
designed for use by students in grades 1-12, that are based on motivation 
theory and provide multiple feedback mechanisms. Research questions included: 
What motivation theory or theories are most appropriate as a basis for the 
comprehensive evaluation of Web sites? and Do the WebMAC instruments 
adequately and validly test the motivational quality of Web sites? A survey 
of the motivational literature was conducted to investigate the first 
question. A comprehensive, well -researched theory that could be adapted to 
the assessment of electronic environments (particularly Web sites) was 
sought. Formative evaluation methods that included iterative testing of the 
instruments were implemented with graduate and undergraduate students, 
educators, and children, while factor analyses were conducted to confirm the 
theoretical basis of the instrument. These provided evidence for the second 
question. (Contains 10 references.) (MES) 
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The evaluation of Web resources has become a critical information skill for students. Because of 
the uniqueness of these resources, new tools are required that not only assess the content validity 
and mechanical functionality of a Web site, but also its motivational quality. This paper 
describes the development and validation of a set of Web evaluation tools (Website Motivational 
Analysis Checklist: WebMAC), designed for use by students in grades 1-12, that are based on 
motivation theory and provide multiple feedback mechanisms. 



Introduction 



to 



rv 




( 

v 



Just as with the growth of print resources, as the number of electronic resources continues to 
escalate, the need to evaluate those resources increases in importance. The evaluation of 
networked information resources, such as Web sites, requires new tools that encompass a 
broader concept of evaluation, In addition to the appropriateness and authenticity of content and 
soundness of the design, other evaluation criteria include such issues as currency and coverage 
(Alexander & Tate, 1998). Furthermore, assessing the functionality of the various mechanisms of 
a site to identify technical problems and improve user control and Web site consistency is 
important (Nielsen, 1994). 

A number of evaluation instruments (e.g. Caywood, 1998; Schrock, 1997) have been developed 
to help library media specialists and teachers judge the suitability of a Web site for their 
instructional needs. Some focus heavily on content validity issues while others focus on 
functionality issues. While these issues are extremely important, there is another issue that has 
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largely been ignored by other instruments; i.e. the motivational quality of a Web site. 
Motivational quality refers to the site's ability to motivate users to visit, engage, and return to it. 
Furthermore, most Web site evaluation instruments are designed for use by adults; few are 
created for independent use by students or as a tool for teaching students a structured method for 
evaluating Web sites. Finally, few instruments have detailed scoring mechanisms that allow 
visual analysis and interpretation for improving the site. 

Until now, there have been no Web site evaluation instruments that both focus on motivational 
quality (how well the site attracts and engages the user) and are designed for use by both students 
and educators. The Website Motivational Analysis Checklist (WebMAC)© is a series of 7 
instruments, five of which are designed for use in education contexts. WebMAC Senior (Small & 
Amone, 1998), a 32-item instrument, was designed to be used with students in grades 9 and up. 
All other instruments were adapted from WebMAC Senior. WebMAC Junior (Amone & Small, 
1999), a 16-item instrument is targeted toward students in grades 1-4. Two other versions of 
WebMAC Junior, WebMAC Investigator (Amone & Small, 1999) a 12-item instrument and 
WebMAC Junior— Long Form (Amone & Small, 1998), a 24-item instrument, are also available 
for this level. WebMAC Middle (Small & Amone, 1998), a 24-item instrument, is intended for 
students in grades 5-8; 

All of these instruments emphasize motivational quality while including functionality and 
content validity items, framed in terms of their effect on motivation. These instruments differ 
from other Web site evaluation instruments because they are (1) theoretically based, (2) user- 
centered, (3) use a research approach, and (4) allow feedback for improvement from multiple 
viewpoints (Nielsen, 1994). 



Research Questions 

This evaluation study documents the creation and testing of these Web site evaluation 
instruments and describes their potential applications in instruction. Research questions include: 

1. What motivation theory(ies) are most appropriate as a basis for the comprehensive 
evaluation of Web sites? 

2. Do the WebMAC instruments adequately and validly test the motivational quality of Web 
sites? . 



Methodology 

A survey of the motivational literature was conducted to investigate Research Question #1. A 
comprehensive, well-researched theory that could be adapted to the assessment of electronic 
environments (particularly Web sites) was sought. Formative evaluation methods that included 
iterative testing of the instruments were implemented with educators and children while factor 
analyses were conducted to confirm the theoretical basis of the instrument. These provided 
evidence for Research Question #2. 
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Results 



Research Question #/. 

md eared ^ l0ng Standin §’ ^-researched theory that 

Expectancy value (F VUh apP ! 1Cab !? t0 and com Prehensive for the evaluation of Web sites. 
Expectancy-value (E-V) theory (e.g. Vroom, 1964) states that motivation is defined by the type 

^f u “ e ef ° "l? ind f iVidUal ° n 3 taSk ‘ ft forther S P edfies *"> Prerequisites^ effort! 
result. expectation for success. Both prerequisites must be met for effort (motivation) to 

While E-V theory has been widely supported in research on motivation in the work environment 
t is only recently that this theory has been applied to research on 

T 1 ™™ 1' , F ° r exam P le > Burton et al. (1992-1003) and Snead and Harrell (1994) found E-V 

Ind 7f a . U l th ;° retlCal frameWOrk for stud y in § motivation in evaluation of expert ^sterns 

and of electronic decision support systems. P systems 

f t Vdle0ry the theoretical basis for the WebMAC instruments described in this paper- i e 
foe degree t° which both value and expectation for success are attained comprises the 
motivational quality of a Web site. Value is defined in terms of two attributes— how stimulating 

ofirr^ the Web Si !T iS t0 *** USer ‘ Ex P ectation for Success is also defined in term! 
of two attributes— how organized and how easy-to-use the Web site is for the user. 

The WebMAC instruments use Likert-type items (strongly agree-strongly disagree- not 

toaDDrom° a « eValU f m fh a , WCb S ‘‘ e ' " N °‘ Applkable ” items ™ further scored in relation to 
tneir appropnateness for that site. 

Examples of Value and Expectation for Success items from WebMAC Senior are: 

9. This Web site is fun and interesting to explore. (Value: Stimulating) 

16. Navigating this Web site does not require any special skills or experience. 

(.expectation for Success: Easy-to-Use) 

Z e a^ C A-*?J ddle US6S the Same format aS WebMAC Se n'or but uses simpler language and 
deXl y t dlff ® rent ratmgS ( defimtel y agme - do NOT agree). There is no "Not Applicable” 
designation. Examples of Value and Expectation for Success items from WebMAC Middle are: 

19. The information at this Web site is useful to me. (Value: Meaningful) 

is , a “7'“ map that helps mc understand how much and what kinds of 
information I will find there. (Expectation for Success: Organized) 

i?do^tJfh “h” 8 smi f ey / aces ( excclIent - VCI >' poor) to represent response choices. 

It does not break items down mto the four attributes but simply categorizes items by Value and 

Junior are- 0 ^ ^ UCCess - exam P^ e of Value and Expectation for Success items from WebMAC 
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1. When you first arrived at this Web site, did it look like this would be an interesting or 



fun Web site to explore? (Value) 
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Was it easy to find your way around 
for Success) 


the different parts of this Web site? (Expectation 
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After completing the scoring of all items on a WebMAC instrument, the student transfers scores 
to the appropriate E-V category columns and totals scores. Totaled scores are plotted on scoring 
graphs and grids for quick visual assessment of the strengths and weaknesses of the site and 
comparison among evaluations. 



Research Question #2: 

More than 100 students and educators have participated in iterative testing and validation of 
these instruments during the past two years. Initial testing focused on WebMAC Senior since it 
was the first instrument developed and the other instruments were adapted from it. 

A group of 23 graduate students was asked to independently evaluate an assigned Web site using 
a 60-item WebMAC Senior. Items were randomly ordered to prevent clustering and 
categorization. Analysis focused on the distribution of item scores. Items with a wide spread of 
scores (set at 1.00+ standard deviation) were either revised or eliminated, resulting in a 40-item 
instrument. 

Two additional tests of the 40-item WebMAC Senior were conducted in which 34 graduate and 
undergraduate students were observed as they evaluated one of two pre-selected Web sites that 
were considered moderately motivating. They provided qualitative feedback on each item and on 
the overall instrument. Finally, the instrument was administered to 226 graduate and 
undergraduate students who were asked to assess the same Web site. Factor analyses were 
conducted to verify clustering of items into proposed categories, resulting in the final 32-item 
instrument. 

The original WebMAC Junior (24-item) and WebMAC Middle (32-item) instruments were 
reviewed by 21 educators for clarity, comprehensibility, and face validity, resulting in minor 
modifications. WebMAC Junior was subsequently tested with ten children in a formative 
evaluation conducted by the producers of a nationally-televised children's television program. 
Information about individual items was collected to determine how children understood and 
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interpreted each question. Additional observations and focus group interviews yielded rich 
instalment ^ ^ Pr ° V ' ded 2 m ° re in ' depth understanding of children's attitudes toward the 

Finally a large-scale pilot test was conducted on WebMAC Junior, with over 500 students 

(grades 1-4) participating in testing the instrument. As a result, the wording of several items was 

modified and the shorter, 16-item instrument was created. Factor analyses on the results will be 

WpbM^uvP/r reported , ,® a ^ ed on the results of testing the other WebMAC instruments, 
yveb MAC Middle was modified to a 24-item instrument. 

The WebMAC instruments are currently being used by educators around the world. WebMAC 
Aemor has been translated into Portuguese for use in Brazil and WebiVIAC Junior has been 
translated into Japanese and Portuguese. In addition, the instrument has been used in more than 
40 schools and colleges across the United States. Complete background information 
descriptions, the instruments, their scoring and administration directions, and stories of 
implementation in a variety of contexts are contained in our books, "Motivation Mining- Using 
Eva uation Ski Is to Find Web Treasures(Grades 1-6) and "Motivation Mining: Using Evaluation 
Skills to Find Web Treasures (Grades 7-12), available this fall from Linworth Publishing, Inc. 
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